Switching Text-Based Image Encoders for Captioning Images With Text

Visual understanding, such as image caption generation, has received extensive attention. Describing images with textual information is one way to help people achieve barrier-free visibility. This study focuses on the text-based captioning (TextCaps) task. The TextCaps task more complex than traditional because it depends optical character recognition (OCR) and that appears in image. It also requires consideration of relationship between recognized objects OCR’s linguistic part In this study, we propose maximizing use multiple modalities an improve performance. We enrich OCR features using pre-trained Contrastive Language-Image Pre-training (CLIP) models. then introduce two additional attention models a transformer architecture strengthen representation modality. experimental results demonstrate our proposed method, which introduces multimodal four image-related modalities, outperforms existing methods for dataset.